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Abstract — The public key cryptosystem based on 
rank error correcting codes (the GPT cryptosystem) 
was proposed in 1991. Use of rank codes in crypto- 
graphic applications is advantageous since it is prac- 
tically impossible to utilize combinatoric decoding. 
This enabled using public keys of a smaller size. 
Several attacks against this system were published, 
including Gibson's attacks and more recently Over- 
beck's attacks. A few modifications were proposed 
withstanding Gibson's attack but at least one of them 
was broken by the stronger attacks by Overbeck. 
A tool to prevent Overbeck's attack is presented in 
I12I . In this paper, we apply this approach to other 
variants of the GPT cryptosystem. 

I. Introduction 

The first code-based public -key cryptosystem is 
introduced and investigated in [1|. The system is 
based on Goppa codes in the Hamming metric. It 
is a strong cryptosystem but the size of a public 
key is too large for practical implementations to 
be efficient. 

The public key cryptosystem based on rank error 
correcting codes was proposed in lIU, Js] and is 
now called the GPT cryptosystem. 

Rank codes are well structured. It makes eas- 
ier creation of attacks. Subsequently in a series 
of works, Gibson jU, (|5| developed attacks that 
break the GPT system for public keys of about 
5 Kbits which are efficient for practical values of 
parameters n < 30, where n is length of rank codes 
with the field ¥2'^ as an alphabet. 

Several variants of the GPT PKC were intro- 
duced to withstand Gibson's attacks ||6l, Q. One 
proposal is use of a rectangular row scramble ma- 
trix instead of a square matrix. This allows to work 
with subcodes of rank codes having much more 
complicated structure. Another proposal exploits a 
modification of Maximum Rank Distance (MRD) 
codes where the concept of a column scramble ma- 



trix was also introduced. A new class of rank codes, 
so called, reducible codes, are also implemented to 
modify the GPT cryptosystem iS], All these 
variants withstand Gibson's attack. 

Recently, R. Overbeck lITol . ifTTl proposed a 
new attack which is more effective than any of 
Gibson's attacks. His method is based on the fact 
that a column scrambler is defined over the base 
field. A generalization and development of one 
Gibson's idea allows him to break many instances 
of the GPT cryptosystem. It was found in ||T21 
that a cryptographer can define a proper column 
scrambler over the extension field without violation 
of the standard mode of the PKC. It turns out that 
Overbeck's attack fails in this case. 

In this paper, we implement an idea of a proper 
choice of column scramblers over the extension 
field to other variants of the GPT cryptosystem. 
This choice withstands Overbeck's attacks as well 
as Gibson's attacks. 

II. The GPT cryptosystem 
A. Rank codes 

Let ¥q be a finite field of q elements and let F^w 
be an extension field of degree N. 

Let X = {xi,X2, ■ ■ ■ ,Xn) be a vector with 
coordinates in F^jv. 

The Rank norm of x is denoted Rk(x | F^) and 
is defined as the maximal number of Xi, which are 
linearly independent over the base field ¥q. 

Similarly, for a matrix M with entries in F^n 
the column rank is defined as the maximal number 
of columns, which are linearly independent over 
the base field F,, and is denoted Rk(M | F,). 

The Rank distance between x and y is defined 
as the rank norm of the difference x — y, i.e. 
d(x,y)=Rk(x-y|F,). 



The theory of optimal MRD (Maximal Rank 
Distance) codes is given in [13], A generator 
matrix Gfc of a MRD code is defined by 
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where gi , 52 , • • ■ , 9n are any set of elements of the 
extension field F,™ which are linearly independent 
over the base field F^. 

The notation g[*l g'^ means the ith 

Frobenius power of g. 

A code with the generator matrix ([T]) is referred 
to as a {n,k,d) code, where n is the code length, 
k is the number of information symbols, d is the 
code distance. For MRD codes, d = n — k + 1. 

Let m = {mi, m2, . . . , rrifc) be an information 
vector of dimension k. The corresponding code 
vector is the n-vector 

g(m) = mGfe. 

If y = g(m) + e and Rk(e) = s <t = 
then the information vector m can be recovered 
uniquely from y by some decoding algorithm. 

There exist fast decoding algorithms for MRD 
codes liT3l . lfT4l . A decoding procedure requires 
elements of the (n — k) x n parity check matrix H 
such that GfeH^ = 0. For decoding, the matrix H 
should be of the form 
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where elements hi, h2, ■ ■ ■ , hn are in the extension 
field ¥qn and are linearly independent over the base 
field F,. 

B. Description of the standard GPT cryptosystem 

The GPT cryptosystem is described as follows. 

1) Possible generator matrices using as public 
keys: Denote by Gpub the public key, which is a 
generator matrix of a code. 

1) 

Gp„b - SGfcP. (3) 

The main matrix Gfc is given by Eq. ([1]). It 
is used to correct rank errors. Errors of rank 
not greater than t ~ can be corrected. 



A square k y. k matrix S over the extension 
field Wqn is called the row scrambling matrix. 
It is used to destroy any visible structure of 
the matrix Gfc by mixing its rows. 
A matrix P — \pij\ is called the column 
scrambler This matrix is a non singular 
square matrix of order n. It is used to mix 
columns of Gfc. 

If P is a matrix over the base field ¥q, then 
a matrix GfcP has just the same structure 
as the matrix Gfc with a different first row. 
Hence, from the point of view of breaking, 
matrices Gpub = SGfcP and Gpub — SGfc 
are equivalent. A cryptographer may not use 
a matrix P at all. 

On the other hand, if entries pij are in the 
extension field Fg^, then a matrix P makes 
breaking much harder. We shall analyze this 
case. 

2) Another generator matrix is obtained by an 
extension of matrix Gfc: 



■"pub 



S[X Gfc.] p. 



(4) 



A matrix X of size kxti is called a distortion 
source matrix. This matrix is a part of the 
concatenation [X Gfc] . The column rank 
of X is Rk(X I ¥q) = ti. The number ti is a 
design parameter. Another design parameter 
is the ordinary rank which can take values 
from 1 to ti. The rank distance of a code 
generated by the matrix Gpub is not less than 
the rank distance of a code generated by the 
mati-ix S [O Gfc] P. 

A matrix P is called the column scrambler 
This matrix is a non-singular square matrix 
of order n + ti. It is used to mix and to 
corrupt columns of Gfc by means of the 
distortion source matrix X. 
Note that in previous works, the matrix P has 
all its entries in the base field ¥q. Overbeck's 
attack against this PKC succeeded due to this 
fact. But the attack fails for the proper choice 
of P over the extension field F„i. 11^. 



3) 



Gpub-S[X Gfc.] P. 



(5) 



4) 



Here a scrambling matrix S is a rectangular 
{k — p) X k matrix. 



Gpub = S(^[0 Gfc] + [Xi X2]jP. (6) 

Here: the row scrambler S is a square non- 
singular matrix of order k with entries in Fq^ 
chosen at random; O is the k x m matrix 



of O's; Xi is some k x m matrix — the 
first distortion matrix; X2 is a fc x n matrix 
with r(X2|Fi) — ti — the second distortion 
matrix; the column scrambler P is a non- 
singular matrix of order n + m with entries 
inF,. 

2) Plaintext: For public keys (O, (|4|l and (O, a 
plaintext is any fc-vector m = {mi, m2, . . . , mk), 
rus G Fqi. , s = 1, 2, . . . , fc. For the public key Q, 
a plaintext is a (fc — p)-vector. 

3) Private keys: The Private keys are matrices 
S, Gk, X, , Xi, X2, P separately and (explicitly) a 
fast decoding algorithm of an MRD code. Note 
also, that the matrices X, Xi,X2 are not used 
to decrypt a ciphertext and can be deleted after 
calculating the Public key. 

4) Encryption: Let m — {mi, 7712, . . . , mk) be 
a plaintext. The corresponding ciphertext is given 
by 

c = mGp„b + e = mS[X|Gfe]P + e, (7) 

where e is an artificial vector of errors of rank t2 
or less, randomly chosen and added by the sending 
party. The number t2 is the third design parameter. 

5) Decryption: The legitimate receiver upon re- 
ceiving c calculates 

c' = cP"^ = mS[X|Gfe] + eP-\ 

Then he extracts from c' the plaintext m using 
decoding algorithms and properties of public keys. 

III. The Overbeck attack - an idea 

In iFTOl . ifm . a new attack is proposed on the 
GPT PKC described by means of Eq. dH). 

It is claimed, that similar attacks can be proposed 
on all the variants of GPT PKC. 

We can not describe the attack in detail but recall 
briefly an idea of this attack. 

We need some notations. 



For X e F„., let a : F„ 



F„ 



a{x) 



be the Frobenius automorphism. 

For flie matrix T = {tij) over F^n, let o-(T) = 
= {t%). 

For any integer s, let cr^(T) = cr(cr^~i(T)). 
It is clear that cr" = a. Thus the inverse exists 

The following simple properties of cr are useful; 

• a{a + b) = a{a) + cr(6). 

• a{ab) — a{a)a{b). 

• In general, for matrices cr(T) ^ T. 

• If P is a matrix over the base field Fg, then 
a(P) = P. 



6) An idea of Overbeck's attack: To break a 
system, a cryptanalyst constructs from the public 
key Gpub = S [X G^] P the extended public 
key as follows: 



G 



ext^pub 



Gp 
a{Gp 



(8) 



The property that cr(P) = P, if P is a matrix 
over the base field Fg, is used in (|8]l. Further 
transformations of Eq. ^ allows to obtain the first 
row of the check matrix H of the rank code used. 
It is enough to break the cryptosystem. 

If P is a matrix over the extension field F^n, 
then cr(P) ^ P. 

We liave to stress tliat Overbeck's attack fails 
in this case. 

Moreover Gibson's attacks use also in implicit 
form the condition (t(P) = P and can not be 
implemented without it. 

Our intention is to show that there exist column 
scramblers P in the extension field F^n such that 
the GPT PKC works and is secure against all 
known attacks. 

IV. Other attacks on the GPT PKC 

An important part of a decryption procedure 
is correcting rank errors using a fast decoding 
algorithm known to the legitimate party. An unau- 
thorized party may want to correct rank errors by 
a general algorithm without any knowledge of the 
structure of a rank code. We consider algorithms 
described in |15J and in the recent paper 116|. 

The authors of ifTSl proposed two algorithms for 
decoding an arbitrary {n, fc) linear rank distance 
code over F^n. These algorithms correct errors 
of rank t = [^\ in O ((Aft)3g(*-i)(fc+i)) and 
O ((fc + operations in F, respec- 

tively. 

Consider as an example a case when we use a 
(28, 14) rank code with N = n = 28, k = U,q = 
2,d = 15, t = 7. The size of the public key is equal 
to Nnk — 10976 bits. To correct 7-fold rank errors, 
Ourivski-Johansson's algorithms |15| require 2^^^^ 
and 2^"^^ operations in F2. Thus these attacks are 
infeasible for practical implementations. 

The algorithm of |16| requires 
O (log((7)iV'^(^^*)) operations. We have for the 
above example 2'''^^ operations. Thus this attack 
is also infeasible for practical implementations. 



V. The simple GPT PKC 

Consider the public key of Eq. No distortion 
matrix X is used. A ciphertext has the form 



mSGiP + e, 



(9) 



where the rank Rk(e | ¥q) = ti of an artificial 
error e is less or equal to t = L^V^J ■ 

Brute-force attacks are based on the exhaustive 
search of possible artificial errors e. It depends on 
the number of error vectors. If artificial errors are 
all possible n-vectors of rank ii, then the number 
of operations to search is O (q"*!). 

Attacks on the public key contemplate to find 
unknown factors (to a cryptanalyst) S, and 
P, or, to find matrices S, Gfe and P such that 
SGfeP = SGfcP from the known public key 
matrix SG^P . 

Assume first that the column scrambler P is a 
matrix over the base field F,. The legitimate user 
knows the secret key P and P^^. His algorithm is 
as follows. 

1) Get a ciphertext c = mSGfcP + e. 

2) Multiply to the right by P"\ Get an inter- 
mediate ciphertext 

-1 



cP 



mSGi + eP- 



(10) 



Note that Rk(eP-i | F,) 
[^J since P- 



= Rk(e I F,) = 
^ is in the base 



h < t 
field ¥q. 

3) Decode c' using a fast decoding algorithm 
and get mS. 

4) Get a plaintext m as (mS)S^^. 

On the other hand, the cryptanalyst can get a 
successful representation Gpub — SGfc for the 
equivalent rank code with the generator matrix Gfe 
from the public key SGfcP. It can be done by 
means of Gibson-Overbeck's attacks and therefore 
break the system. 

The situation is quite different if P is a matrix 
over the extension field F^w . For the general matrix 
P, it is unknown how to solve the following 
problems: to find the public key factors S, G^, 
and P, or, to find matrices S, Gfc and P such 
that SGfeP = SGfcP from the known public key 
matrix SG^P. Gibson-Overbeck's attacks are not 
applicable if a matrix P is chosen in the extension 
field F,iv. 

We can assume from now on that Gibson's and 
Overbeck's attacks can not be implemented. But 
the cryptographer should select a secret column 
scrambler P in the extension field F^n and a public 
set £ of artificial errors e such that 

n — k 



where eP~^ is an error in the intermediate cipher- 
text ( [Tol l. 

a) Choice of 8,: The public set of artificial 
errors is chosen as the set consisting of all n- 
vectors in F^V with rank ti < t: 

£ = {e:eeF';«,Rk(e|F,)=ti}. 

b) Choice of F: The cryptographer chooses 
an inverse matrix P^^ in the form P^^ = 
[Qi Q2], where Qi is a submatrix of size 
n X [t — <i) with entries in the extension field F^n 
while Q2 is a submatrix of size n x {n — t + ti) 
with entries in the base field ¥q. 

Lemma 1: Let e be any n-vector of rank ti. 
Then the condition Eq. (fTTI) is hold. 

Proof: We have eP^^ = e [Qi Q2] = 
[eQi eQ2] . A vector e can be represented as 
e = [wi W2 ... W(J A, where w^-'s are 
linearly independent over and A is the ti x 
n matrix over F^ of rank ti. Then eQi = 
[wi W2 ... J Bi, where Bi = AQ^ is 
the ti X (t — ti) matrix over the extension field 
FqN. It is clear that Rk(eQi \¥q) <t - ti. Sim- 
ilarly, eQ2 = [wi W2 ... wtj B2, where 
B2 = AQ2 is the ti X {n — t + ti) matrix over 
the base field F,. It follows that Rk(eQ2 | F,) = 
min(ii, n — t + ti) < ti. Hence 

Rk(eP-i I Fq) < Rk(eQi | F,) + Rk(eQ2 | F,) 

■ 

Remark 1: The matrix P^^ can be replaced by 
a matrix P^^ = P^^Q, where Q is any nxn non 
singular matrix over the base field F,). 

Example 1: Consider again the case when we 
use a (28, 14) rank code with N = n = 28,k = 
14:, q = 2,d = 15, t — 7. Possible systems are 
listed below. 

= 0, P in the extension field, attacks on PK - 
Information sets attacks, brute-force attacks - not 
needed, status - insecure. 

ti ~ 1, P in the extension field, attacks on 



PK - unknown, brute-force attacks 



status 



Rk(eP"^ \ ¥q) <t- 



(11) 



- msecure. 

ti = 2, P in the extension field, attacks on 
PK - unknown, brute-force attacks - 2"**, status 

- insecure. 

= 3, P in the extension field, attacks on PK - 
unknown, brute-force attacks - 2^^, status - secure. 

ti = 4, P in the extension field, attacks on PK - 
unknown, brute-force attacks - 2^^, status - secure. 

ti = 5, P in the extension field, attacks on 
PK - unknown, brute-force attacks - 2^^", status - 
secure. 



ti — 6, P in the extension field, attacks on 
PK - unknown, brute-force attacks - 2^*"^, status - 
secure. 

ti — 7, P in the base field, attacks on PK 
- Gibson-Overbeck, brute-force attacks - 2^^*, 
status - insecure. 

For ti = ... 2, the system is insecure due 
to brute-force attacks. For ti — 7, the system 
is insecure because of Gibson-Overbeck's attacks 
since in this case the matrix P is in the base field 
¥q. But for ti = 3... 6, the system is secure 
against all known attacks. We recommend the value 
ti = 3, or the value ti = 4. 

VI. Other variants of the GPT PKC 

We can repeat word for word all previous con- 
siderations for variants (|4|i- ^ and choose for 
each case a proper column scrambler P over the 
extension field F^n . This prevents Overbeck's and 
Gibson's attacks. 

VII. Conclusion 

An approach is presented to withstand attacks 
on the GPT Public key cryptosystem based on rank 
codes. 

It is shown that there exist column scramblers P 
over the extension field W^n which allow decryp- 
tion for the authorized party while an unauthorized 
party can not break the system by means of known 
attacks. 
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